Type C-[ instead. This should send ASCII code 27 just like an Escape key would. C-3 may also work on some terminal (but not under X). For many terminals (notably DEC terminals) <F11> generates <ESC>. If not, the following form can be used to bind it:
;; F11 is the documented ESC replacement on DEC terminals.
(define-key function-key-map [f11] [?\e])